Papers with conversational settings
Ask to Learn: A Study on Curiosity-driven Question Generation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing work on Question Generation focuses on generating relevant questions given text with an answer . human ability to ask questions goes beyond evaluation of reading comprehension . |
| Approach: | They propose a novel text generation task based on a conversational question-asking dataset . they investigate automated metrics to measure different properties of Curious Questions . |
| Outcome: | The proposed task is based on a conversational Question Answering dataset . the results show that humans tend to ask questions with the goal of obtaining new information . |
Recognising Agreement and Disagreement between Stances with Reason Comparing Networks (P19-1)
Copied to clipboard
| Challenge: | Existing methods for (dis)agreement detection focus on conversational settings . however, non-dialogic stance-bearing utterances are common in real-world scenarios . |
| Approach: | They propose a reason comparing network to leverage reason information for stance comparison. |
| Outcome: | The proposed method outperforms baselines on a well-known stance corpus. |
SCAI-QReCC Shared Task on Conversational Question Answering (2022.lrec-1)
Copied to clipboard
| Challenge: | evaluating systems for conversational QA remains an open research problem in its own right . evaluating (conversational) QA systems remains an important challenge for developing conversational information retrieval (conversional search) systems. |
| Approach: | They propose to use a conversational question answering task to extend the original conversational QA dataset with alternative correct answers produced by participant systems. |
| Outcome: | The proposed task was based on the SCAI-QReCC 2021 shared task on conversational question answering. |
Language Models in Dialogue: Conversational Maxims for Human-AI Interactions (2024.findings-emnlp)
Copied to clipboard
Erik Miehling, Manish Nagireddy, Prasanna Sattigeri, Elizabeth Daly, David Piorkowski, John Richards
| Challenge: | Modern language models exhibit some inherent shortcomings, particularly in conversational settings. |
| Approach: | They propose a set of maxims for describing effective human-AI conversation that include quantity, quality, relevance, manner, benevolence, and transparency. |
| Outcome: | The proposed maxims are applied to human-AI interactions and are based on extensive research from the social science and AI communities. |
ASTRID - An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systems (2025.findings-acl)
Copied to clipboard
| Challenge: | Current automated RAG metrics perform poorly in clinical and conversational use cases. |
| Approach: | They propose an automated and scaleable TRIaD for evaluating clinical QA systems leveraging Retrieval Augmented Generation (RAG) metric consisting of three metrics: Context Relevance (CR), Refusal Accuracy (RA), and Conversational Faithfulness (CF). |
| Outcome: | The proposed metric captures the faithfulness of a model’s response without penalising conversational elements and captures refusal to address questions outside of the system’s scope of practice. |
TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that conversational Large Language Models (LLMs) can perform ill-defined complex tasks with different prompt types/styles and different degrees of detail. |
| Approach: | They propose a general taxonomy that can be used to design prompts with specific properties to perform a wide range of complex tasks. |
| Outcome: | The proposed taxonomy will allow future benchmarking studies to report specific categories of prompts used as part of the study, enabling meaningful comparisons across different studies. |